Skip to content

DEV-1408: Keep the callback consumer thread alive when a handler throws - #344

Merged
MAzalya merged 1 commit into
masterfrom
DEV-1408_consumer_thread_survives_exception
Nov 4, 2026
Merged

MAzalya merged 1 commit into
masterfrom
DEV-1408_consumer_thread_survives_exception

Conversation

@MAzalya

@MAzalya MAzalya commented Nov 4, 2026

Copy link
Copy Markdown
Contributor

Problem

BasicProcessWithCallBack.ConsumerThread caught only InterruptedException. A RuntimeException thrown by any processMsgFromCallback() implementation ended the consumer thread for good, and nothing reported it (under javaw the stack trace goes nowhere). After that:

  • the queue (LinkedBlockingDeque, capacity 1024) stopped draining;
  • once it was full, every producer blocked in put(), and so did every producer upstream of those;
  • anything that later sent a callback hung, including an application's shutdown hook. The JVM then kept running with no window.

This was found in a Consensys process that had stayed alive like this for five days after its window was closed. A thread dump showed two queues full at exactly 1024 messages, with no consumer left on the top-level one.

Fix

The consumer now handles each message through processMsgSafely(), which catches a RuntimeException, logs the thread name, message identifier and stack trace to System.err, and carries on with the next message. Errors are still not caught.

Tests

  • New BasicProcessWithCallBackTest: a handler throws on the first message, and the test checks that the later messages are still processed and the thread is still alive. It fails on the unpatched class ("message after the exception was never processed") and passes with the fix.
  • ShimmerDriver gradlew test: 178 tests, 0 failures.

Reaching consumers

This needs a driver release and pin bumps before ShimmerAndroidAPI and ASM_PC (submodule) pick it up. Consensys builds the driver from source at its release tag.

Other host APIs: the queue/consumer-thread pattern exists only in this Java driver. The C#, Python, TypeScript, Swift and Rust implementations have no equivalent, so nothing to port.

A companion change in the Consensys API bounds its shutdown hook with a timeout, so that a stuck cleanup can no longer keep the process alive.

🤖 Generated with Claude Code

ConsumerThread caught only InterruptedException, so a RuntimeException
from processMsgFromCallback() ended the thread for good. Its queue then
stopped draining, producers blocked in put() once it held 1024
messages, and anything that later sent a callback hung - including an
application's shutdown hook, which left the JVM running with no window.

Each message is now processed in processMsgSafely(), which logs a
RuntimeException and lets the thread carry on.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

@jyong15 jyong15 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good

@MAzalya
MAzalya merged commit 483e63c into master Nov 4, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants